Closing The Loop On Test Creation Information Theoretic Feedback Mechanisms For Instructors

New accreditation requirements focus on education as a “continuous improvement process.” The most important part of such a process is that information gets fed back into the system to improve the quality of output. This requirement is often interpreted to mean a feedback loop that iterates on offerings of courses or entire academic years. This paper provides two smaller and more immediate feedback loops based on information theory. These techniques give instructors feedback on the quality of each question on a test or quiz, as well as a numeric score for the instrument of assessment itself. A tool implementing these can be used to help instructors learn which questions are difficult, as well as what types of questions are correlated with ability, and how to design a meaningful instrument of assessment. Comparing these measurements against the percentage of students responding correctly shows a clear benefit for these new techniques.